Technical Field
This disclosure relates to the capture of high resolution facial geometry and reflectance.
Description of Related Art
Modeling realistic human characters is frequently done using 3D recordings of the shape and appearance of real people across a set of different facial expressions to build blendshape facial models. See PIGHIN, F., HECKER, J., LISCHINSKI, D., SZELISKI, R., AND SALESIN, D. H. 1998, Synthesizing realistic facial expressions from photographs, In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, N.Y., USA, SIGGRAPH '98, 75-84; ALEXANDER, O., ROGERS, M., LAMBETH, W., CHIANG, J.-Y., MA, W.-C., WANG, C.-C., AND DEBEVEC, P. 2010, The Digital Emily Project: Achieving a photoreal digital actor, IEEE Computer Graphics and Applications 30 (July), 20-31. Believable characters which cross the “Uncanny Valley” require high-quality geometry, texture maps, reflectance properties, and surface detail at the level of skin pores and fine wrinkles. Unfortunately, there does not yet appear to have been a technique for recording such datasets which is near instantaneous and relatively low-cost.
While some facial capture techniques are instantaneous and inexpensive, see BEELER, T., BICKEL, B., BEARDSLEY, P., SUMNER, B., AND GROSS, M. 2010; High-quality single-shot capture of facial geometry, ACM Trans. Graph. 29 (July), 40:1-40:9; BRADLEY, D., HEIDRICH, W., POPA, T., AND SHEFFER, A. 2010; High resolution passive facial performance capture, ACM Trans. Graph. 29 (July), 41:1-41:10, these may not provide lighting-independent texture maps, specular reflectance information, and/or high-resolution surface normal detail for relighting. In contrast, techniques which use multiple photographs from spherical lighting setups, see WEYRICH, T., MATUSIK, W., PFISTER, H., BICKEL, B., DONNER, C., TU, C., MCANDLESS, J., LEE, J., NGAN, A., JENSEN, H. W., AND GROSS, M. 2006, Analysis of human faces using a measurement-based skin reflectance model, ACM TOG 25, 3, 1013-1024; GHOSH, A., FYFFE, G., TUNWATTANAPONG, B., BUSCH, J., YU, X., AND DEBEVEC, P. 2011, Multiview face capture using polarized spherical gradient illumination, ACM Trans, Graphics (Proc. SIGGRAPH Asia) 30, 6, may capture such reflectance properties, but may come at the expense of longer capture times and complicated custom equipment. More Detailed Description of Various Approaches
Passive Multi-View Stereo
There is a rich history of work in the computer vision literature on passive multi-view stereo reconstruction of scenes including faces. FURUKAWA, Y., AND PONCE, J. 2009, Dense 3D motion capture or human faces, In Proc. of CVPR 09, proposed multi-view stereopsis as a match-expand filter procedure that produces dense patch reconstruction from an initial set of sparse correspondences. However, since subsurface scattering typically blurs surface detail, see RAMELLA-ROMAN, J. C. 2008, Out of plane polarimetric imaging of skin: Surface and subsurface effect, In Optical Waveguide Sensing and Imaging, W. J. Bock, I. Gannot, and S. Taney, Eds., NATO Science for Peace and Security Series B: Physics and Bio541 physics. Springer Netherlands, 259-269, “10.1007/978-1-4020-542 6952-9_12”, for semi-translucent materials such as skin, the resolution which can be recovered for faces may be limited.
Passive multi-view stereo has been employed by Beeler et al. 2010 and Bradley et al. 2010, supra, to reconstruct high quality facial geometry under diffuse illumination. Beeler et al. apply mesoscopic augmentation as in GLENCROSS, M., WARD, G. J., MELENDEZ, F., JAY, C., LIU, J., AND HUBBOLD, R. 2008, A perceptually validated model for surface depth hallucination, ACM Trans Graph 27, 3 (August), 59:1-59:8, to hallucinate detailed geometry, which, while not metrically accurate, may increase the perceived realism of the models by adding the appearance of skin detail.
Valgaerts et al. present a passive facial capture system which achieves high quality facial geometry reconstruction under arbitrary uncontrolled illumination. VALGAERTS, L., W U, C., BRUHN, A., SEIDEL, H.-P., AND THEOBALT, C. 2012, Lightweight binocular facial performance capture under uncontrolled lighting, ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012) 31, 6 (November), 187:1-187:11. They reconstruct base geometry from stereo correspondence and incorporate high frequency surface detail using shape from shading and incident illumination estimation as in Wu et al. W U, C., VARANASI, K., LIU, Y., SEIDEL, H.-P., AND THEOBALT, C. 2011, Shading-based dynamic shape refinement from multi-view video under general illumination, In Proceedings of the 2011 International Conference on Computer Vision, ICCV '11, 1108-1115. The technique may achieve impressive results for uncontrolled lighting, but may not take full advantage of specular surface reflections to estimate detailed facial geometry and reflectance.
Structured Lighting Systems
Numerous successful techniques using structured light projection have addressed 3D facial scanning, including applications to dynamic facial capture, see RUSINKIEWICZ, S., HALL-HOLT, O., AND LEVOY, M. 2002, Real-time 3D model acquisition, ACM TOG 21, 3, 438-446; ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M. 2004, Spacetime faces: high resolution capture for modeling and animation, ACM TOG 23, 3, 548-558. However, these techniques may operate at a lower resolution than may be needed to record high resolution facial detail and may not specifically address reflectance capture.
Diffuse Photometric Stereo Photometric Stereo,
WOODHAM, R. J. 1978, Photometric stereo: A reflectance technique for determining surface orientation from image intensity, In Proc. SPIE's 22nd Annual Technical Symposium, vol. 155, has been applied to recover dynamic facial performances using simultaneous illumination from a set of red, green and blue lights, see HERNANDEZ, C., VOGIATZIS, G., BROSTOW, G. J., STENGER, B., AND CIPOLLA, R. 2007. Non-rigid photometric stereo with colored lights. In Proc. IEEE International Conference on Computer Vision, 1-8; KLAUDINY, M., HILTON, A., AND EDGE, J. 2010, High-detail 3D capture of facial performance, In International Symposium 3D Data Processing, Visualization and Transmission (3DPVT). However, these techniques may either be data intensive or may not recover reflectance information. An exception may be Georghiades, see GEORGHIADES, A. 2003, Recovering 3-D shape and reflectance from a small number of photographs, In Rendering Techniques, 230-240, who recovers shape and both diffuse and specular reflectance information for a face lit by multiple unknown point lights. The problem is formulated as uncalibrated photometric stereo and a constant specular roughness parameter is estimated over the face, achieving a medium scale reconstruction of the facial geometry. ZICKLER, T., MALLICK, S. P., KRIEGMAN, D. J., AND BELHUMEUR, P. N. 2008, Color subspaces as photometric invariants, Int. J. Comput. Vision 79, 1 (August), 13-30, showed that photometric invariants allow photometric stereo to operate on specular surfaces when the illuminant color is known. The practicality of photometric surface orientations in computer graphics has been demonstrated by RUSHMEIER, H., TAUBIN, G., AND GUÉZIEC, A. 1997, Applying shape from lighting variation to bump map capture, In Rendering Techniques, 35-44, for creating bump maps, and NEHAB, D., RUSINKIEWICZ, S., DAVIS, J., AND RAMAMOORTHI, R. 2005, Efficiently combining positions and normals for precise 3D geometry, ACM TOG 24, 3, 536-54, for embossing such surface orientations for improved 3D geometric models. HERTZMANN, A., AND SEITZ, S. M. 2005, Example-based photometric stereo: Shape reconstruction with general, varying brdfs, PAMI 27, 8, 1254-1264, showed that with exemplar reflectance properties, photometric stereo can be applied accurately to materials with complex BRDF's, and GOLDMAN, D. B., CURLESS, B., HERTZMANN, A., AND SEITZ, S. M. 2005, Shape and spatially-varying brdfs from photometric stereo, In ICCV, 341-348, presented simultaneous estimation of normals and a set of material BRDFs. However, all of these may require multiple lighting conditions per viewpoint, which may be prohibitive to acquire using near-instant capture with commodity DSLRs.
Specular Photometric Stereo
Most of the above techniques have exploited diffuse surface reflectance for surface shape recovery. This is because typically specular highlights may not be view-independent and may shift across the subject as the location of the light and camera changes. ZICKLER, T. E., BELHUMEUR, P. N., AND KRIEGMAN, D. J. 2002, Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction, Int. J, Comput Vision 49,2-3, 215-227, exploits Helmholtz reciprocity to overcome this limitation for pairs of cameras and light sources. Significant work, see CHEN, T., GOESELE, M., AND SEIDEL, H. P. 2006, Mesostructure from specularities, In CVPR, 1825-1832; WEYRICH et al, supra; and DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., SAROKIN, W., AND SAGAR, M. 2000, Acquiring the reflectance field of a human face, In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., New York, N.Y., USA, SIGGRAPH '00, 145-156, analyzes specular reflections to provide higher-resolution surface orientations for translucent surfaces. MA, W.-C., HAWKINS, T., PEERS, P., CHABERT, C.-F., WEISS, M., AND DEBEVEC, P. 2007, Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination, In Rendering Techniques, 183-194; and GHOSH et al, supra, perform photometric stereo using spherical gradient illumination and polarization difference imaging to isolate specular reflections, recording specular surface detail from a small number of images. While these techniques can produce high quality facial geometry, they may require a complex acquisition setup such as an LED sphere and many photographs.
Diffuse-Specular Separation
Both polarization and color space analysis can be used in separating diffuse and specular reflections, see NAYAR, S., FANG, X., AND BOULT, T. 1997, Separation of reflection components using color and polarization, IJCV 21, 3, 163-186. MALLICK, S. P., ZICKLER, T. E., KRIEGMAN, D. J., AND BELHUMEUR, P. N. 2005, Beyond lambert: Reconstructing specular surfaces using color, In CVPR, use a linear transform from RGB color space to an SUV color space where S corresponds to the intensity of monochromatic specular reflectance and UV correspond to the orthogonal chroma of the diffuse reflectance